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DESCRIPTION 

3D image segmentation 

The present invention relates to the field of digital imaging and, more particularly, to 
the field of 3D image segmentation by the use of deformable models. In detail, the 
present invention relates to a method of segmenting a three-dimensional structure of 
interest, to an image processing device and to a computer program for an image 
processing unit 

Segmentation methods are used to derive three-dimensional structures of interest, such 
as, for example, organs or bones from volumetric image data, such as CT, MR or US 
images. One popular segmentation approach is the use of deformable models. A survey 
about deformable models has been, for example, published by Mclherney et al.: 
"Deformable models in medical image analysis: A survey", Medical Image Analysis, 1 
(2): pp. 91-108, 1996. A deformable model can be represented as an elastic surface, the 
shape and position of which can change under the influence of an internal energy and an 
external energy. The internal energy serves to preserve as well as possible the shape of 
the model (which may have been formed on the basis of prior knowledge concerning the 
structure to be segmented). The external energy, however, should move the model 
surface in the direction of the object's edges. The external energy is derived from a 
three-dimensional image, that is from a dataset, which is a three-dimensional 
representation of the object containing the structure. Such a three-dimensional 
representation of the object usually consists of a plurality of two-dimensional images, 
each representing a slice of the object. The meshes are preferably shaped as triangles, 
the comer points of which are defined by three neighboring network points on the 
surface of the model. 
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According to the known method, structure points are searched on the surface of the 
structure to be segmented, i.e. along search lines, which may extend perpendicular to 
the triangular surfaces in the image. The surface of the structure to be segmented is 
usually characterized by a pronounced gradient of the image data. After the 
determination of such structure points for the meshes of the network defining the 
surface of the model, the network points of the model are calculated anew on the basis 
of the structure points found, i.e. the network points are calculated in such a manner that 
the weighted sum of the internal energy and the external energy assumes a minimum 
value for the newly found network points. According to this method, the position of the 
network points is influenced, not only by the directly neighboring structure points found 
during the search, but also by all other structure points. Subsequently, the method is 
repeated iteratively a number of times, by utilizing the newly calculated network points. 
Thus, the model is deformed upon each iteration, and the shape assumed by the model 
after the last iteration, i.e. after a shut-off criterion has been reached, is considered to be 
the structure to be segmented in the concrete object. 

The three-dimensional image of the object may be generated, for example, during 
medical examinations, by means of the x-ray computed tomography method, the 
magnetic resonance method, or the ultrasonic method. 

In other words, according to the above method, the deformable model fitting process is 
guided by minimization of the sum of the external energy, based on image feature 
information, which attracts the mesh to the boundaries of the structure of interest, and 
an internal energy, which preserves the consistent shape of the mesh. 

A problem that may occur, for example, during the segmentation of the structure of 
interest using CT image data, is that the three-dimensional image often does not contain 
reliable feature information, for example, image gradients, to attract the mesh, In other 
words, it may occur that in some of the two-dimensional slices of the three-dimensional 
image, the surface of the structure of interest is not " visible" to the segmentation 
process, due to, for example, an insufficient gradient of the image data to attract the 
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mesh. This may appear for several reasons: a poor discrimination of soft tissue, for 
example, in CT, a low image resolution or artifacts due to metal implants. 

It is an object of the present invention to provide for an improved segmentation. 

According to an exemplary embodiment of the present invention, the above object may 
be solved with a method of segmenting a three-dimensional structure of interest 
containing an object from a plurality of two-dimensional images in accordance with 
claim 1. According to this exemplary embodiment of the present invention, images of 
the plurality of two-dimensional images with insufficient feature information for the 
structure of interest are identified Then, attractors in the form of at least a partial 
contour of the structure of interest are manually drawn in the images of the plurality of 
two-dimensional images with insufficient feature information. Then, the structure of 
interest is segmented in the plurality of images by using, for example, the segmentation 
algorithm described above, with the manually drawn attractors replacing structure 
points in the areas of two-dimensional images where feature information is insufficient. 
Advantageously, according to an aspect of the present invention, this method may 
enable a more reliable fitting of 3D deformable models to structures, such as organ 
parts, where poor contrast, noise or image artifacts do not allow for a fully automatic 
boundary delineation. Furthermore, this may advantageously improve a robustness of 
the segmentation process. 

Furthermore, in instances where, for example, from a hundred two-dimensional images 
forming the three-dimensional image, only, for example, twenty contain insufficient 
feature information for the structure of interest, according to the present invention the 
manual placement or drawing of the attractors has only to be performed in these twenty 
images and not in all hundred images. Thus, the interaction required from a user is 
minimize d, which, in particular in clinical applications, is a critical concern. 

According to another exemplary embodiment of the present invention as set forth in 
claim 2, the identification of two-dimensional images with insufficient feature 
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information includes a segmentation process. In other words, according to the present 
invention, the segmentation process is applied to the three-dimensional image. Then, on 
the basis of the segmentation results from this first segmentation process, images of the 
two-dimensional images are identified containing insufficient feature information. Then, 
for these images with insufficient feature information, the manually drawn attractors are 
added. Then, a second segmentation process is performed, in which the attractors are 
included in the images previously identified as having insufficient feature information. 

Advantageously, according to this exemplary embodiment of the present invention, the 
same segmentation process used for the final segmentation may be used for 
identification of the images containing insufficient feature information. 

According to another exemplary embodiment of the present invention as set forth in 
claim 3, the segmentation of the structure of interest in the plurality of images is based 
on an iterative optimization of internal and external energy. Such structure segmentation 
based on an energy optimization is described in further detail by Weese et al.: "Shape 
constrained deformable models for 3D medical image segmentation" Proc. of 17 th 
International Conference on Information Processing in Medical Imaging (IPMT), pages 
380 to 387, Davies, CA, USA, 2001, Springer-Verlag, which is hereby incorporated by 
reference. 

Advantageously, the method according to this exemplary embodiment allows for a fast, 
efficient and very accurate segmentation process, which is also robust. 

According to another exemplary embodiment of the present invention as set forth in 
claim 4, the segmentation for finally segmenting the structure of interest is also based 
on energy minimization, as, for example, described by Weese et al.: "Shape constrained 
deformable models for 3D medical image segmentation", Proc. of 17 th International 
Conference on Information Processing in Medical Imaging (IPMT), pages 380 to 387, 
Davies, CA, USA, 2001, Springer- Verlag, which is hereby incorporated by reference. 
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According to another exemplary embodiment of the present invention as set forth in 
claim 5, the manual drawing of attractors in the form of at least a partial contour of the 
structure of interest in the images of the plurality of two-dimensional images including 
insufficient feature information, involves an assignment of labels to voxels 
corresponding to the at least partial contour, such that the attractors are considered as 
strong edges of the structure of interest during segmentation. Thus, according to this 
exemplary embodiment of the present invention, the attractors are labeled such that they 
simulate a surface boundary of the structure of interest, i.e. a pronounced gradient in the 
image data. Thus, the method according to the present invention may, without further 
adaptation, be applied to a plurality of segmentation methods. 

According to another exemplary embodiment of the present invention as set forth in 
claim 6, the method is for radiation therapy planning (RIP) on the basis of CT images. 
Advantageously, this exemplary embodiment of the present invention allows for an 
improved radiation therapy planning, due to the fest and accurate segmentation of the 
structure of interest 

According to another exemplary embodiment of the present invention as set forth in 
claim 7, an image processing device is provided, comprising a memory for storing the 
images, a pointer for drawing the attractors, and an image processor for segmenting the 
three-dimensional structure of interest in the images. According to an aspect of the 
present invention, attractors are manually drawn, which may correspond to at least a 
partial contour of the boundaries of the structure of interest in images with insufficient 
feature information. Due to this, an improved segmentation is provided, even if there are 
images with insufficient feature information. 

According to another exemplary embodiment of the present invention as set forth in 
claim 8, a computer program is provided for an image processing unit, for segmenting a 
three-dimensional structure of interest from a plurality of two-dimensional images. The 
computer program may be written in any suitable programming language, for example, 
C++ and may, for example, be stored on a computer readable medium, such as a CD- 
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ROM. Also, the computer program may be available from a network such as the World 
Wide Web, from which it may be downloaded to an image processing unit or suitable 
computer. The computer program, according to this exemplary embodiment of the 
present invention, performs the method according to the present invention as set forth in 
claim 1. 

It may be seen as the gist of an exemplary embodiment of the present invention that 
images of the two-dimensional images forming a three-dimensional image of an object 
are identified, containing insufficient feature information for an automatic delineation or 
segmentation of the three-dimensional structure of interest in the object Then, 
according to an aspect of the present invention, in these images, attractors are drawn, 
which may substantially follow at least a partial contour of the structure of interest This 
may be done by an operator. The attractors are such that they can be easily 
discriminated by a feature search algorithm of a segmentation process, for example, 
through assignment of specific labels outside the typical intensity range of the image to 
voxels corresponding to the manually drawn contours. In the deformable model fitting 
process or segmentation process, the manually drawn contours may then be considered 
as artificial strong edges, which attract the model. 

These and other aspects of the present invention will become apparent from and 
elucidated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the following, 
with reference to the following drawings: 

Fig. 1 shows a schematic representation of an image processing device according to an 
exemplary embodiment of the present invention, adapted to execute an exemplary 
embodiment of a method according to the present invention. 

Fig. 2 shows a flowchart of an exemplary embodiment of a method according to the 
present invention. 
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Fig. 3 shows a two-dimensional image of the 3D image, depicting the bladder. 

Fig. 4 shows the image of Fig. 3, where attractors have been added to the bladder in 
accordance with the present invention and the model is initialized 

Fig. 5 shows the image of Fig. 4, where the model is adapted to the attractors in 
accordance with the present invention. 

Fig. 1 shows an exemplary embodiment of an image processing device according to the 
present invention. The image processing device shown in Fig. 1 includes an image 
processing and control processor 1 with a memory 2 in which a deformable model of a 
structure to be segmented can be stored and which is capable of storing a plurality of 
two-dimensional images forming a three-dimensional image of an object to be 
examined The image processing and control processor may be coupled, via bus system 
3, to an imaging device (not shown in Fig. 1), for example, an MR apparatus, a US 
(ultrasonic) apparatus, or aCT apparatus. The structure segmented by the image 
processing and control processor can be displayed on a monitor 4. The user can access 
the image processing and control processor 1 via a keyboard 5 or other input means, 
which are not shown in Fig. 1, such as a mouse or a trackball. The keyboard 5, the 
mouse or the trackball are all considered as pointers, by which an operator may perform 
a pointing operation on an image displayed on the monitor 4 to, for example, draw 
attractors in the images displayed on the monitor A. 

Fig. 2 shows a simplified flowchart of a method or a computer program according an 
exemplary embodiment of the present invention for operating the image processing 
device depicted in Fig. 1. In Ihe following example, CT images are used However, it 
has to be noted that, as indicated above, the method of the present invention may also be 
applied to other suitable images. After the start in step SI , the method loads a plurality 
of two-dimensional CT images forming a three-dimensional image of an object from, 
for example, a CT scanner in step S2. Then, the method continues to step S3, where an 
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automatic segmentation process is performed on the plurality of two-dimensional CT 
images in order to determine images of the plurality of two-dimensional images with 
insufficient feature information for the structure of interest The segmentation process or 
deformable model fitting process is, as will be explained in further detail in the 
following, guided by minimization of the sum of an external energy, based on image 
feature information, i.e. a pronounced gradient designating a surface of the structure of 
interest, which attracts the mesh to the boundaries of the structure of interest and an 
internal energy, which preserves the consistent shape of the mesh. A problem in the 
segmentation process may occur if the two-dimensional images in the three-dimensional 
image do not contain reliable feature information, for example, image gradients, to 
attract the mesh. This may happen due, for example, to a poor discrimination of soft 
tissue, for example, in CT, to a low image resolution of the two-dimensional images, 
artifacts due to implants, such as, for example, metal implants or the like. 

In the segmentation process performed in step S3, a deformable model M, for example, 
of the bladder, is adapted to the structure of interest The surface of the model is formed 
by a network, the triangular meshes of which connect three neighboring network points 
on the surface of the model to one another three times. 

Instead of a model thus subdivided into triangles, the surface of the model could also be 
defined by a polygonal structure, in which the meshes of the network do not have a 
triangular shape, but link a number of network points other than three to one another in 
the form of a polygon or a simplex. 

The deformable model is usually represented by a mesh consisting of V vertices with 
coordinates x. and T triangles. To adapt the mesh to the structure of interest in the two- 
dimensional^^ 

surface detection step and a mesh reconfiguration step. Mesh reconfiguration is done by 
minimizing: 
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E=E^ +aR 



'int 



(1) 



The external energy Eext drives the mesh towards the surface patches obtained in the 
surface detection step. The internal energy E^t restricts the flexibility of the mesh. The 
parameter a weights the relative influence of each term- The different components of 
the algorithm are now described in the following: 

Surface detection 

For surface detection, a search is performed along a triangle normal ni to find a point % 
with the optimal combination of feature value and the distance 8j to the triangle 

center x. : 

x.=x t + n^argmax^CJc, + ntf) -DS 2 j 2 } (2) 

y=-/, a 

The parameter / defines the search profile length, the parameter 8 is the distance 
between two successive points, and the parameter D controls the weighting of the 
distance information and the feature value. For example, the quantity 

F t (xy-±n.gix) ^ Hg0e ^ ™ 

may be used as a feature, where g(x) denotes the image gradient at point x . The sign is 
chosen in dependence on the brightness of the structure of interest, with respect to the 
surrounding structures. For image points with a gradient magnitude smaller than a 
threshold , this quantity is essentially the gradient in the direction of the mesh 
normal. 
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During segmentation, this threshold prevents problems that occur, if the structure of 
interest has a considerably smaller gradient magnitude at the boundary than other 
objects in the neighborhood, or if different parts of the object of interest have 
boundaries with considerably different gradient magnitude. 

External energy 

In analogy to iterative closest point algorithms, the external energy 

(4) 

may be used, where the weights w ( have been introduced to give the more promising 
surface points jc, a larger influence during mesh reconfiguration. With this external 
energy, the detected surface points would directly attract the triangle centers of the 
mesh. As a consequence, once a triangle center has been attracted by a surface point in 
the image, it can hardly move anymore. For this reason, the mesh remains attached to 
false object boundaries, which are detected frequently at the beginning of the adaptation 
process. This problem is diminished considerably if the triangle centers x t are attracted 
by the planes perpendicular to the image gradient at the surface point 3^. : 

As may be gathered from the above equations, the external energy is based on a distance 
between the deformable model or shape model after adaptation and feature points, i.e. a 
boundary of the structure of interest. 
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Internal energy 

The starting point for the introduction of the internal energy is a shape model 
represented by a mesh of triangles with vertex coordinates 

M 

m { = m? + y *T J p k mt; i=l,...,F. (6) 

In this equation, m* 0 denote the vertex coordinates of the mean model, rof describe the 
variation of the vertex coordinates associated with the M eigenmodes of the model, and 
p k represent the weights of the eigenmodes. 

Since the shape model provides a suitable distribution of mesh vertices, the internal 
energy has been designed to main tain this distribution. For that purpose, the difference 
vectors between the coordinates of two neighboring mesh vertices are considered. 
Difference vectors for the deformable model and the shape model are compared, and the 
deviations between both are penalized: 

where the set N(i) contains the neighbors of vertex i . The scale s and the orientation R 
of the shape model, as well as its weights p k , have to be determined in addition to the 
vertex coordinates x t during the mesh reconfiguration. 

The segmentation process, i.e. the mesh reconfiguration is then performed by 
minimization of the total energy E as set forth in equation (1). 

This method is described in further detail by Weese et al.: "Shape constained 
deformable models for 3D medical image segmentation", Proc. of 17 th International 
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Conference on information Processing in Medical Imaging (IPMT), pages 380 to 387, 
Davies, CA, USA, 2001, Springer-Verlag, which is hereby incorporated by reference. 

Then, in the subsequent step S4, images of the plurality of two-dimensional images, 
such as CT images, where the automatic segmentation failed, are identified- However, 
an image containing insufficient feature information may also be identified by an 
operator in a way that it is displayed to the operator on the monitor 4, who may then 
decide. In case the image does not contain sufficient feature information, i.e. the 
automatic segmentation in accordance with the method described with reference to step 
S3 feils, in step S5, attractors may be manually drawn by an operator in these images 
identified in step S4. For this, these images with insufficient feature information may be 
displayed to the operator who places the attractors by means of the pointer. Preferably, 
these attractors correspond to at least a partial contour of the structure of interest. As 
already mentioned, the manually drawn attractors may have the form of complete or 
partial contours corresponding to the boundaries of the structure of interest. These 
attractors are such that they can be easily discriminated by the feature search algorithm, 
which is part of the segmentation process performed in the subsequent step S6. This 
may be ensured, for example, through the assignment of specific labels outside the 
typical intensity range for the image data to the voxels corresponding to the manually 
drawn contours. In other words, by the manually drawn attractors, the boundaries or 
portions of the structure of interest may be characterized by an artificially pronounced 
gradient of the image data. Then, in the subsequent segmentation process or deformable 
model fitting process, the manually drawn contours or attractors may be considered as 
artificial strong edges, which attract the model in the absence of reliable image feature 
information. 

For the manual drawing of attractors in these edges, these images may be individually 
displayed to the operator via the monitor 4. 

In the subsequent step S6, an automatic segmentation process is performed on the 
plurality of two-dimensional images, such as CT images, by using the attractors. The 
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method used as the automatic segmentation method, is preferably the same as described 
with reference to step S3. Thus, in order to avoid unnecessary repetition, reference is 
made to step S3 for the description of the segmentation process. In step S6, due to the 
fact that in the images with insufficient feature information with respect to the structure 
of interest, the attractors "simulate" an artificial strong edge, or a pronounced gradient, 
the segmentation is successful in all of the plurality of two-dimensional images, even in 
the images with insufficient feature information. Then, in the subsequent step S7, the 
segmentation results obtained in step S6 are output, for example, via the monitor 4 and 
then the method continues to step S8, where it ends. 

Advantageously, the above described method allows to significantly reduce the amount 
of interaction required from an operator, since the attractors do not have to be added to 
all of the plurality of two-dimensional images, but only to those containing insufficient 
feature information, where the automatic segmentation feils. Furthermore, 
advantageously, the present invention allows to improve the robustness of automated 
organ delineation tools, for example, in RTP systems. Furthermore, in particular in 
radiation therapy planning, improvements can be achieved, since the segmentation 
results achieved with the present invention are highly accurate and allow for an accurate 
radiation therapy planning. Also, due to the semi-automatic interaction by the operator, 
a highly reliable setting of the deformable models to the structure of interest, such as 
organ parts, can be achieved, even where poor contrast noise or image artifacts do not 
allow a fully automatic boundary delineation. 

Fig. 3 shows a two-dimensional CT image depicting the hip region of a male patient 
Reference numerals 10 and 12 designate femur heads and reference numeral 13 
designates the pelvis. Reference numeral 14 designates the spinal cord. Between two 
pelvis portions 16 and 18, the bladder 20 is located. As may be taken from Fig. 3, the 
bladder 20 is almost indistinguishable and there are no pronounced gradients in the 
image data of Fig. 3, clearly identifying boundaries of the bladder 20. 
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Fig. 4 shows the image of Fig. 3, where the contour 22 of the bladder 20 was marked 
manually by, for example, an operator tracing the boundary or contour of the bladder 20 
with a pointer device on the monitor 4, such that the voxels in the image of Fig. 4 are 
labeled accordingly. As may be taken from Fig. 4, the contour 22 is completely white, 
and thus represents a strong edge, which can easily be segmented by the segmentation 
process, since it strongly attracts the model. The starting model to be adapted to the 
contour 22 of the bladder 20 is indicated in Fig. 3 by reference numeral 24. As may be 
taken from Fig. 4, at the beginning of the segmentation process, the model 24 is not 
closely adapted to the contour 22 of the bladder 20. 

Fig. 5 shows the image of Fig. 4 at the end of the segmentation process, when the model 
24 is closely matched to the contour 22. 

Accordingly, as may be taken from a comparison of Figs. 3 and 5, even in cases where 
the feature information included in the image is not sufficient to clearly distinguish 
boundaries of a structure of interest, such as the bladder 20, the present invention allows 
a highly accurate semi-automatic segmentation of such structure of interest 
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CLAIMS 

1 . Method of segmenting a three-dimensional structure of interest which is 
contained in an object from a plurality of two-dimensional images, wherein each of the 
plurality of images represents a slice of the object, the method comprising the steps of: 

determining images of the plurality of two-dimensional images with insufficient 
feature information for the structure of interest; 

manually drawing attractors in the form at least a partial contour of the structure 
of interest in the images of the plurality of two-dimensional images; and 

segmenting the structure of interest in the plurality of images by using the 
attractors. 

2. Method according to claim 1 , 

wherein the determination of images of the plurality of two-dimensional images 
with insufficient feature information for the structure of interest comprises the steps of: 

segmenting the structure of interest in the plurality of images; 

identifying the images of the plurality of two-dimensional images with 
insufficient feature information for the structure of interest on the basis of at least one 
decision criterion. 

3. Method according to claim 2, 

wherein the segmentation of the structure of interest in the plurality of images is 
based on an adaptation of a deformable model whose surface is formed by a network of 
meshes which interconnect network points on the surface of the deformable model to 
the structure of interest; and 

wherein the adaptation of the deformable model is based on an iterative 
optimization of an internal energy based on a distance between the deformable model 
after adaptation and the deformable model and an external energy basedon a distance 
between the deformable model after adaptation and feature points on the structure of 
interest 
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4. Method according to claim 1 , 

wherein the segmentation of the structure of interest in the plurality of images by 
using the at least partial contour of the structure of interest is based on an adaptation of 
a deformable model whose surface is formed by a network of meshes which 
interconnect network points on the surface of the deformable model to the structure of 
interest; and 

wherein the adaptation of the deformable model is based on an iterative 
optimization of an internal energy based on a distance between the deformable model 
after adaptation and the deformable model and an external energy based on a distance 
between the deformable model after adaptation and feature points on the structure of 
interest. 

5. Method according to claim 1, 

wherein the manual drawing of attractors in the form of at least a partial contour 
of the structure of interest in the images of the plurality of two-dimensional images 
includes an assignment of labels to voxels corresponding to the at least partial contour 
such that the attractors are considered as strong edges of the structure of interest during 
segmentation. 

6. Method according to claim 1, 

wherein the method is for radiotherapy planning on the basis of CT images. 

7. Image processing device, comprising: 

a memory for storing a plurality of two-dimensional images wherein each of the 

plurality of images represents a slice of an object; 

a pointer for drawing attractors in the plurality of two-dimensional images; and 
an image processor for segmenting a three-dimensional structure of interest 

which is contained in the object from the plurality of two-dimensional images, which 

segmentation operation is performed as follows: 



-17- 



PHDE030206 EP-P 



determining images of the plurality of two-dimensional images with insufficient 
feature information for the structure of interest; 

manually drawing attractors in the form of at least a partial contour of the 
structure of interest in the images of the plurality of two-dimensional images; and 

segmenting the structure of interest in the plurality of images by using the 
attractors. 

8. A computer program for an image processing unit for segmenting a three- 
dimensional structure of interest which is contained in an object from a plurality of two- 
dimensional images, wherein each of the plurality of images represents a slice of the 
object, which computer program comprises the steps of: 

determining images of the plurality of two-dimensional images with insufficient 
feature information for the structure of interest; 

manually drawing attractors in the form of at least a partial contour of the 
structure of interest in the images of the plurality of two-dimensional images; and 

segmenting the structure of interest in the plurality of images by using the 
attractors. 
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ABSTRACT 

3D image segmentation 

A delineation of a structure of interest can be performed by fitting 3D defonnable 
models, for example, represented by polygonal measures, to the boundaries of the 
structure of interest. The defonnable model fitting process is guided by minimization of 
the sum of an external energy, based on image feature information, which attracts the 
mesh to the organ boundaries and an internal energy, which preserves the consistent 
shape of the mesh. A frequent problem is that the images do not contain sufficient 
reliable image feature information, such as image gradients, to attract the mesh. 
According to the present invention, manually drawn attractors in the form of complete 
or partial contours corresponding to boundaries of the structure of interest are placed 
into the images which do not contain sufficient feature information. These attractors 
may easily be discriminated by a subsequent segmentation process. Due to this, 
advantageously, a 3D defonnable model can be fitted to structures of interest in images 
with poor contrast, noise or image artifacts. 



(Fig. 3) 
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